{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Otto Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以Kaggle 2015年举办的Otto Group Product Classification Challenge竞赛数据为例，进行XGBoost参数调优探索。\n",
    "\n",
    "竞赛官网：https://www.kaggle.com/c/otto-group-product-classification-challenge/data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第四步：调整树的参数：subsample 和 colsample_bytree\n",
    "(粗调，参数的步长为0.1；下一步是在粗调最佳参数周围，将步长降为0.05，进行精细调整)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先 import 必要的模块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "train = pd.read_csv(dpath +\"Otto_train.csv\")\n",
    "#train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variable Identification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "选择该数据集是因为的数据特征单一，我们可以在特征工程方面少做些工作，集中精力放在参数调优上"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Target 分布，看看各类样本分布是否均衡"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#sns.countplot(train.target);\n",
    "#pyplot.xlabel('target');\n",
    "#pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每类样本分布不是很均匀，所以交叉验证时也考虑各类样本按比例抽取"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop ids and get labels\n",
    "y_train = train['target']\n",
    "y_train = y_train.map(lambda s: s[6:])\n",
    "y_train = y_train.map(lambda s: int(s)-1)\n",
    "\n",
    "train = train.drop([\"id\", \"target\"], axis=1)\n",
    "X_train = np.array(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各类样本不均衡，交叉验证是采用StratifiedKFold，在每折采样时各类样本按比例采样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第二轮参数调整得到的n_estimators最优值（645），其余参数继续默认值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用交叉验证评价模型性能时，用scoring参数定义评价指标。评价指标是越高越好，因此用一些损失函数当评价指标时，需要再加负号，如neg_log_loss，neg_mean_squared_error 详见sklearn文档：http://scikit-learn.org/stable/modules/model_evaluation.html#log-loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'colsample_bytree': [0.6, 0.7, 0.8, 0.9],\n",
       " 'subsample': [0.3, 0.4, 0.5, 0.6, 0.7, 0.8]}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "subsample = [i/10.0 for i in range(3,9)]\n",
    "colsample_bytree = [i/10.0 for i in range(6,10)]\n",
    "param_test3_1 = dict(subsample=subsample, colsample_bytree=colsample_bytree)\n",
    "param_test3_1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Applications/anaconda/lib/python2.7/site-packages/sklearn/model_selection/_search.py:667: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -1.85464, std: 0.00674, params: {'subsample': 0.3, 'colsample_bytree': 0.6},\n",
       "  mean: -1.85163, std: 0.00646, params: {'subsample': 0.4, 'colsample_bytree': 0.6},\n",
       "  mean: -1.85072, std: 0.00696, params: {'subsample': 0.5, 'colsample_bytree': 0.6},\n",
       "  mean: -1.84962, std: 0.00677, params: {'subsample': 0.6, 'colsample_bytree': 0.6},\n",
       "  mean: -1.84909, std: 0.00615, params: {'subsample': 0.7, 'colsample_bytree': 0.6},\n",
       "  mean: -1.84797, std: 0.00602, params: {'subsample': 0.8, 'colsample_bytree': 0.6},\n",
       "  mean: -1.84590, std: 0.00294, params: {'subsample': 0.3, 'colsample_bytree': 0.7},\n",
       "  mean: -1.84309, std: 0.00218, params: {'subsample': 0.4, 'colsample_bytree': 0.7},\n",
       "  mean: -1.84174, std: 0.00263, params: {'subsample': 0.5, 'colsample_bytree': 0.7},\n",
       "  mean: -1.83981, std: 0.00291, params: {'subsample': 0.6, 'colsample_bytree': 0.7},\n",
       "  mean: -1.83898, std: 0.00334, params: {'subsample': 0.7, 'colsample_bytree': 0.7},\n",
       "  mean: -1.83723, std: 0.00243, params: {'subsample': 0.8, 'colsample_bytree': 0.7},\n",
       "  mean: -1.84136, std: 0.00769, params: {'subsample': 0.3, 'colsample_bytree': 0.8},\n",
       "  mean: -1.83782, std: 0.00784, params: {'subsample': 0.4, 'colsample_bytree': 0.8},\n",
       "  mean: -1.83598, std: 0.00724, params: {'subsample': 0.5, 'colsample_bytree': 0.8},\n",
       "  mean: -1.83392, std: 0.00735, params: {'subsample': 0.6, 'colsample_bytree': 0.8},\n",
       "  mean: -1.83312, std: 0.00724, params: {'subsample': 0.7, 'colsample_bytree': 0.8},\n",
       "  mean: -1.83233, std: 0.00712, params: {'subsample': 0.8, 'colsample_bytree': 0.8},\n",
       "  mean: -1.82044, std: 0.00307, params: {'subsample': 0.3, 'colsample_bytree': 0.9},\n",
       "  mean: -1.81751, std: 0.00289, params: {'subsample': 0.4, 'colsample_bytree': 0.9},\n",
       "  mean: -1.81602, std: 0.00365, params: {'subsample': 0.5, 'colsample_bytree': 0.9},\n",
       "  mean: -1.81445, std: 0.00336, params: {'subsample': 0.6, 'colsample_bytree': 0.9},\n",
       "  mean: -1.81375, std: 0.00415, params: {'subsample': 0.7, 'colsample_bytree': 0.9},\n",
       "  mean: -1.81307, std: 0.00302, params: {'subsample': 0.8, 'colsample_bytree': 0.9}],\n",
       " {'colsample_bytree': 0.9, 'subsample': 0.8},\n",
       " -1.8130670687323831)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb3_1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=645,  #第二轮参数调整得到的n_estimators最优值\n",
    "        max_depth=6,\n",
    "        min_child_weight=4,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch3_1 = GridSearchCV(xgb3_1, param_grid = param_test3_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch3_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch3_1.grid_scores_, gsearch3_1.best_params_,     gsearch3_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([ 3.14737277,  3.9220068 ,  4.26192784,  3.97491193,  4.14486356,\n",
       "         4.07868838,  3.60512819,  4.21996498,  4.47547517,  4.59198093,\n",
       "         4.610958  ,  4.48971696,  4.09460516,  4.56710801,  4.94570565,\n",
       "         5.14211259,  4.86743441,  4.96423383,  4.57349935,  5.00916538,\n",
       "         5.33384223,  5.29475842,  5.44691281,  4.88081417]),\n",
       " 'mean_score_time': array([ 0.1820015 ,  0.12046518,  0.11220436,  0.14947243,  0.11784539,\n",
       "         0.14806924,  0.15361562,  0.13415322,  0.16837783,  0.15030785,\n",
       "         0.16234078,  0.15563197,  0.16268759,  0.15575037,  0.15501981,\n",
       "         0.17085381,  0.1632596 ,  0.15656061,  0.13694849,  0.1608706 ,\n",
       "         0.13947201,  0.15271521,  0.13829818,  0.06964984]),\n",
       " 'mean_test_score': array([-1.85463743, -1.85162842, -1.85072196, -1.84962436, -1.849088  ,\n",
       "        -1.8479694 , -1.84590324, -1.84309339, -1.84174369, -1.83980751,\n",
       "        -1.83897909, -1.83722575, -1.84136204, -1.83781705, -1.83597511,\n",
       "        -1.83392389, -1.83311659, -1.83233433, -1.82043788, -1.81751328,\n",
       "        -1.81602086, -1.8144546 , -1.81374651, -1.81306707]),\n",
       " 'mean_train_score': array([-1.85207135, -1.84825608, -1.84669162, -1.84539664, -1.84456935,\n",
       "        -1.84326995, -1.84205857, -1.83832121, -1.83678131, -1.83441364,\n",
       "        -1.83327323, -1.83168846, -1.83776295, -1.83391756, -1.83168831,\n",
       "        -1.8296725 , -1.82855251, -1.82740914, -1.81645463, -1.81354457,\n",
       "        -1.81148741, -1.80991906, -1.80870973, -1.80773672]),\n",
       " 'param_colsample_bytree': masked_array(data = [0.6 0.6 0.6 0.6 0.6 0.6 0.7 0.7 0.7 0.7 0.7 0.7 0.8 0.8 0.8 0.8 0.8 0.8\n",
       "  0.9 0.9 0.9 0.9 0.9 0.9],\n",
       "              mask = [False False False False False False False False False False False False\n",
       "  False False False False False False False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'param_subsample': masked_array(data = [0.3 0.4 0.5 0.6 0.7 0.8 0.3 0.4 0.5 0.6 0.7 0.8 0.3 0.4 0.5 0.6 0.7 0.8\n",
       "  0.3 0.4 0.5 0.6 0.7 0.8],\n",
       "              mask = [False False False False False False False False False False False False\n",
       "  False False False False False False False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'params': ({'colsample_bytree': 0.6, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.8}),\n",
       " 'rank_test_score': array([24, 23, 22, 21, 20, 19, 18, 17, 16, 14, 13, 11, 15, 12, 10,  9,  8,\n",
       "         7,  6,  5,  4,  3,  2,  1], dtype=int32),\n",
       " 'split0_test_score': array([-1.86081631, -1.85671379, -1.85589973, -1.8535873 , -1.85329595,\n",
       "        -1.85172176, -1.8439884 , -1.84091191, -1.8396215 , -1.83685579,\n",
       "        -1.83493681, -1.83523853, -1.84528365, -1.84294395, -1.84008095,\n",
       "        -1.83873445, -1.83702341, -1.83585277, -1.81734901, -1.81431783,\n",
       "        -1.81213302, -1.81074084, -1.80788242, -1.80986465]),\n",
       " 'split0_train_score': array([-1.85773267, -1.85311442, -1.85102012, -1.84884386, -1.84847668,\n",
       "        -1.84662355, -1.83995855, -1.83500959, -1.834267  , -1.8313344 ,\n",
       "        -1.83014369, -1.82884487, -1.84262624, -1.83879383, -1.83526386,\n",
       "        -1.83408134, -1.83231309, -1.8301846 , -1.81258669, -1.80953743,\n",
       "        -1.8066335 , -1.80631207, -1.80341368, -1.80401301]),\n",
       " 'split1_test_score': array([-1.84237423, -1.83926996, -1.83782954, -1.83676881, -1.83760546,\n",
       "        -1.83632862, -1.84193279, -1.84048047, -1.83900711, -1.83642214,\n",
       "        -1.83531631, -1.83348315, -1.82903566, -1.82523413, -1.82460029,\n",
       "        -1.82252419, -1.82211531, -1.82125102, -1.81624467, -1.81379186,\n",
       "        -1.81132171, -1.80996903, -1.80958303, -1.8090304 ]),\n",
       " 'split1_train_score': array([-1.84038902, -1.83690283, -1.8354219 , -1.83384498, -1.83444534,\n",
       "        -1.83269549, -1.83865365, -1.83643289, -1.8345164 , -1.83110153,\n",
       "        -1.83010601, -1.82870338, -1.82649585, -1.82310757, -1.82122015,\n",
       "        -1.81927364, -1.81863452, -1.81738392, -1.81440401, -1.81142229,\n",
       "        -1.80882648, -1.8062192 , -1.80575997, -1.80581224]),\n",
       " 'split2_test_score': array([-1.85500061, -1.8529695 , -1.8518834 , -1.85170482, -1.84977959,\n",
       "        -1.84958963, -1.84632606, -1.84352022, -1.84145009, -1.84068695,\n",
       "        -1.84114686, -1.83953426, -1.84929565, -1.84459953, -1.84338239,\n",
       "        -1.84086759, -1.8400249 , -1.83942371, -1.82382939, -1.82058312,\n",
       "        -1.81908488, -1.81712029, -1.81676221, -1.81528755]),\n",
       " 'split2_train_score': array([-1.85197733, -1.84907402, -1.84735552, -1.84716543, -1.84526199,\n",
       "        -1.84442848, -1.84162103, -1.83870965, -1.83655347, -1.83531802,\n",
       "        -1.83439962, -1.83366223, -1.84423252, -1.83951354, -1.83882854,\n",
       "        -1.83586602, -1.83471356, -1.83316972, -1.82130576, -1.81808452,\n",
       "        -1.81542349, -1.81324682, -1.81197158, -1.80993783]),\n",
       " 'split3_test_score': array([-1.85417236, -1.85227   , -1.85038257, -1.84980709, -1.8494298 ,\n",
       "        -1.84893385, -1.84656879, -1.84415069, -1.84218454, -1.84075578,\n",
       "        -1.84000265, -1.83852283, -1.84732721, -1.84433549, -1.84144042,\n",
       "        -1.83960742, -1.83946933, -1.83840124, -1.82310057, -1.82004397,\n",
       "        -1.82035443, -1.81704008, -1.81789223, -1.81640572]),\n",
       " 'split3_train_score': array([-1.85267236, -1.85021845, -1.84789336, -1.84725822, -1.84612272,\n",
       "        -1.84547976, -1.84555647, -1.84206249, -1.83932692, -1.83729548,\n",
       "        -1.83555339, -1.83433526, -1.84472844, -1.84107652, -1.83804669,\n",
       "        -1.83585913, -1.83535638, -1.83450651, -1.81744777, -1.81442687,\n",
       "        -1.81451724, -1.81160236, -1.81141942, -1.81052526]),\n",
       " 'split4_test_score': array([-1.86082613, -1.85692118, -1.85761754, -1.85625686, -1.85533197,\n",
       "        -1.85327561, -1.85070274, -1.84640555, -1.84645768, -1.84431946,\n",
       "        -1.84349558, -1.83935148, -1.8358662 , -1.83196999, -1.8303695 ,\n",
       "        -1.82788349, -1.82694779, -1.82674096, -1.82166708, -1.81883097,\n",
       "        -1.81721176, -1.81740492, -1.81661511, -1.81474854]),\n",
       " 'split4_train_score': array([-1.85758538, -1.85197067, -1.85176721, -1.84987071, -1.84854004,\n",
       "        -1.84712247, -1.84450313, -1.83939145, -1.83924277, -1.83701878,\n",
       "        -1.83616343, -1.83289655, -1.83073169, -1.82709633, -1.82508231,\n",
       "        -1.82328238, -1.821745  , -1.82180094, -1.81652891, -1.81425175,\n",
       "        -1.81203632, -1.81221483, -1.810984  , -1.80839528]),\n",
       " 'std_fit_time': array([ 0.51723369,  0.19521514,  0.09663428,  0.25921928,  0.14558462,\n",
       "         0.09884328,  0.18654109,  0.11371431,  0.21877208,  0.26381744,\n",
       "         0.12806775,  0.14443049,  0.19681033,  0.2724425 ,  0.16455695,\n",
       "         0.04584316,  0.1377156 ,  0.19330166,  0.15797814,  0.10158864,\n",
       "         0.15645757,  0.06650824,  0.10203499,  0.56893919]),\n",
       " 'std_score_time': array([ 0.06943562,  0.0225352 ,  0.02409093,  0.04749448,  0.03506548,\n",
       "         0.03004994,  0.05483353,  0.05286596,  0.03982301,  0.04520137,\n",
       "         0.0423434 ,  0.03900674,  0.03848011,  0.02913056,  0.04510021,\n",
       "         0.01821715,  0.04991214,  0.03709342,  0.03047587,  0.05716494,\n",
       "         0.02354687,  0.03476269,  0.03753794,  0.02712409]),\n",
       " 'std_test_score': array([ 0.00674126,  0.00646263,  0.00695763,  0.00677241,  0.00615048,\n",
       "         0.00602174,  0.00293537,  0.00218423,  0.00262634,  0.00290514,\n",
       "         0.00334361,  0.00242847,  0.00769494,  0.00783975,  0.0072445 ,\n",
       "         0.00734992,  0.00724476,  0.00712151,  0.00307348,  0.00288541,\n",
       "         0.00365484,  0.00335886,  0.00415304,  0.00301523]),\n",
       " 'std_train_score': array([ 0.00631351,  0.00584496,  0.00588879,  0.00586405,  0.00522344,\n",
       "         0.00536904,  0.00262321,  0.00244238,  0.00219282,  0.00269672,\n",
       "         0.00263236,  0.00242312,  0.00762117,  0.00734478,  0.00717521,\n",
       "         0.0070006 ,  0.00697283,  0.00668157,  0.00295605,  0.00291379,\n",
       "         0.00333511,  0.0030291 ,  0.00346132,  0.00247484])}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch3_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -1.813067 using {'subsample': 0.8, 'colsample_bytree': 0.9}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEFCAYAAAASWssjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8XFX9//HX7JklM9mXrul6gJYd3HDBwrcugAiCAqJf\n9kUr+yLIIloL4hcERUQRvi6gftUvYBH9iRv4BUFAWrCAR0rp3qZJmmSSzD5zf3/cO5OZbJOkk/3z\nfDz6mDtz78yc2yTnfc85955rMwwDIYQQYij2iS6AEEKIyU/CQgghRFESFkIIIYqSsBBCCFGUhIUQ\nQoiinBNdgLHS0tI16tO8Kit9tLdHSlmcSU/2eWaYafs80/YX9n2fa2vLbQO9Li2LATidjokuwriT\nfZ4ZZto+z7T9hbHbZwkLIYQQRUlYCCGEKErCQgghRFESFkIIIYqSsBBCCFGUhIUQQoiiJCyEEEIU\nNW0vyhNCiOkgkzFIJlIkE2kS8TSJ3PLAr1XXBFh2+CxstgGvrRs1CQshhCgxwzDMyjuRJhlPmY+J\nlFWx938tW9H3ey2RIpXMjOi7XW4HS5bX4ykrbfUuYSGEEJgVfCppVuZmZZ3qfUykSVpH8PnLyfyj\n+oIQSI+6HE6XHbfbicvtwF/uxuV24nY7cHvM19weR+41lye7rve1eU3VdHXHSvg/Y5Wr5J8ohBDj\nKJ3O5HXJpHLdMol4mi3uNva29eRV7HndN3ldO9luntHeONThtJuVt9tByO82K3W306rE8yr6Iq+5\n3A7s9n0bSi7zuiQshBDTRzqVyVXqhZV9Klep547W4+mC1/O7bdKpkXXTZNnttlyFXR4qM4/MPY7c\nUX3hEXx+pe7MBUP2/Q7H9D9XSMJCCDFshmFYR/J5ffB5R/PJvKN689HqyukTAIl4ikx6dIfxTpcd\nt8eJx+uiPFSW1z3T213j9piP1dUBYvFkn8reXHY4p38FX0oSFkLMEKlUmu6uOB17I72VeKKw26bf\n0bwVAPF4KrdtJjO6St6spB14vS6CFWW5ytvtdpqVucfZ+1p+APSp6EfSTVNbW05LS9eoyisKSVgI\nMU3EY0m6OuN0hWN0dcbo7ozllrs648SiyVF9brZLxut3E6rM64rJq8TdHmfB0bv52LutWcmX9lRO\nMb4kLISYAgzDIBZNWhW/Wfl3h2OE80IhER/4DByHw0YgWEZ1nZ9QhZeMYfTpsukTAJ7Cs29Kfb6+\nmJokLISYBAzDINKdMIMg1xqI0RWO58JgsPPtXW4HgaCHhjlllIfKKA+aj4Ggh2CoDK/fnavwpVtG\njJaEhRDjIJPJ0B2O0x2O5wVBbyh0d8UHHfD1lDmpqPSZARDy5MIg+89T5pSjfzHmJCyEKIF0KkNX\nOJbXNVQYCj1d8UHP4ff6XdTUBXoDIGiFgrXs9sifqZh48lsoxDAkE+nClkDewHFXOEakOzHg+2w2\n8AU81M8OUd6nVRAIllEe9OB0zbz7RIupR8JCCLJnEsUKzibKD4VYNDXg++x2G4Ggh1nzKvJaBp7c\nsr/cMyMu2BLTn4SFmPYyGYNId5zuLmvMIGx2EyViKVpbuuke6kwip53yoIfahvLe1kBeKPgCHjkl\nVMwIEhZiyovHUnSHY71B0GWeVtptnV7a050Y9EKy7FQPZveQp7CLKFSG1+eSwWMhkLAQk1w6naHH\nahF0h61TSa1gyIbCYK2C7HhBXWM5gaB5Kmm59RgIemhaWENXd0zCQIhhkLAQEyZ7oVn2lNJ+YWC1\nCgbj9jjzKv/ex3Lr0V/uHnJqCK/PTXdPfCx2TYhpR8JCjJlUKj1kEHSH46QGmTHUbrfhL/cwa26o\nIAjyWwdySqkQ40f+2sSoGIZBpCdRUPF39QmCaGTwuYjKvC4qqn0DtAzMMPD63TJwLMQkImEhBpRM\npPq0BPqHwWCDxtm5iKpq/QVhUB7y4C83n7vk2gIhphQJixksncrQ0txF844wiVialuYuurvMIIjH\nBr6uAMAXcFPTECBQ3js+ELCuLQgEPZR55QwiIaYbCYsZJBpJsHt7mN07Otm9o5OWXV2k+8xH5HTZ\nKQ+VUTcraAZBeWEY+AMeuWmMEDOQhMU0ZRgG7W0Rdm/vZPeOMLu3d9LZHs2tt9mgui5Aw+wg9bND\nLFpaSyqdxu2RSemEEP1JWEwTyUSaPbvCZjDs6KR5R7igK8ntcTB3QSUNs0M0zAlR11hecDaRTF0t\nhBiKhMUU1R2O5VoMu3eEaW3uKpjVNFhRxvxF1TTMCdIwO0RljV/OLhJCjJqExRSQyWRo29NjjjVY\nYw7d4d6Lyex2G3WzgjTMNoOhYXYQX8AzgSUWQkw3EhaTUDyWpHlnOBcMzTvDBXdJK/O6aFpSnetS\nqm0I4HTKqahCiLEjYTHBDMMg3BGzupPMLqW9LT0F21TW+HIthoY5IUKVXhmEFkKMKwmLcZa9tiF3\nltKOTqI9vVc6O112Zs2ryI01NMwO4ilzTWCJhRBCwmLMRXoSNO8Y/NoGf7mHxfvXUm8FQ3VdQG6W\nI4SYdCQsSsgwDNpbI9ZAtNlyGPjahhANc4I0zjEnyRNCiMluQsJCKXUScKrW+owB1l0JnAFkgDVa\n60eVUiHgISAIuIErtNbPjWeZB1JwbYMVDol4n2sbFlbRaF34Vj+rHJdb8lkIMfWMe82llLob+BCw\nfoB1FcClwGLAb23zKHAF8Cet9V1KKQX8DDhs3AptKby2oZPW5u6CaxtClV6allTTOCdE/ewgVTV+\nGYgWQkwLE3GY+zfgMeDCAdb1AFswg8KP2boA+CaQvbDACcSKfUllpW+fTietrvLTvCvMtrfb2bZ5\nL9s27yXc0fu1Doed2fMrmdtUxdymSuY0VREon9rXNtTWlk90Ecad7PP0N9P2F8Zmn8csLJRS5wKX\n93n5bK31/yiljh7irduA1wEHcCuA1rrD+swGzO6oy4p9f3t7ZBSlBv3P3WzSrWzfsrfw2gafiwVL\naqifE6RxdoiaPtc2RGMJorHB7+o22c3E6T5kn6e/mba/sO/7PFjQjFlYaK0fAB4Y4ds+AjQCC6zn\nv1dKPau1fkEpdSDwc+AqrfXTJSxqgfUvbGNvaw9VNf7cJHuNc4IEK+TaBiHEzDXZRlvbgSgQ11ob\nSqkOoEIpdQDwS+BTWutXxrIAJ3/mUKqqAnR1F+3pEkKIGWNShIVS6gpgo9Z6rVLqWOB5pVQGeAb4\nA+YYRxlwtzm+TafW+sSxKIvL7aTM65KwEEKIPDbDGPjWmFNdS0vXqHdM+jlnBtnn6W+m7S+UZMxi\nwP52uVRYCCFEURIWQgghipKwEEIIUZSEhRBCiKIkLIQQQhQlYSGEEKIoCQshhBBFSVgIIYQoSsJC\nCCFEURIWQgghipKwEEIIUZSEhRBCiKIkLIQQQhQlYSGEEKIoCQshhBBFSVgIIYQoSsJCCCFEURIW\nQgghipKwEEIIUZSEhRBCiKIkLIQQQhQlYSGEEKIoCQshhBBFSVgIIYQoSsJCCCFEURIWQgghipKw\nEEIIUZSEhRBCiKIkLIQQQhQlYSGEEKIoCQshhBBFSVgIIYQoSsJCCCFEURIWQgghipKwEEIIUZSE\nxTiKx+M8/vhjI3rP+vUvs3HjmyX5/lWrLmDLls0l+axiTjnlBOLx+Ije88AD3+Oxx341rG2ffvov\ntLa2jKZoA3rmmb9y3nmf5cILz2bt2kf7rY9Go3z1qzfxuc+dx/nn/yevv76hZN8txFQwrLBQSjVa\nj+9TSn1eKeUf22JNT3v3to04LJ54Ym1JK8Xp4pe//Bk9PT0l+axUKsW3v30nd955D/fc833Wrn2U\nvXvbCrb56U9/zMKFi7j33h9w7bU3sHXrlpJ8txBThbPYBkqp7wIZpdR3gJ8CTwIrgE+McdnG1C/+\nvJEX/7VnwHUOh4102hjxZx65Xx2fXLF40PU//vGDbN78Ng8++H02bdpIZ2cnAJdddjWLFi1mzZpb\n2L59G/F4nFNPPY2mpoX8/e/P8e9//4umpoU0NDT0+8x4PMaaNbewe/dukskkV1xxDfvtdwBr1tzC\nzp07SKfTnHbapznmmJW597z66nruuecunE4nZWVlrF79dbq7bdx44xfp7u6itbWFk0/+JCeddAqr\nVl3A4sVLefvtt/B6vRx00KG88MJzdHd3c+ed9/DMM0/zf//3FJFIhI6ODs4++zyOPvqY3Hc1N+/m\n9tvXEI/H8HjKuOaa66mv778fWX/961P8+c9/JBaLcdllV9Hd3c3atY+yevXXAbj44nM46aRT2bjx\n36xefRM33vhVbrjhGoLBEO9+91G8611Hcddd38AwDEKhENdddzOBQID77ruHV15ZRyaT4VOf+jQr\nVhzL448/TnPzXpYtO5DZs+cSDAYBOOigg1m/fh0rVhybK9cLLzzPMcf8B1dcsQqfz8+VV147zN8K\nIaaHomEBvAM4ArgZeEBr/WWl1ItjW6zp6bOfPYe33tpILBbj8MPfwUknncK2bVtZs+YW7rjjW6xf\n/zLf+94PsdlsvPDC8+y33/68853v5phjVg4YFACPPfa/NDTM4pZbbmXbtq0899wzaP0GFRUV3HTT\nV4lEejjnnDM5/PB35N7zf//3NCtWHMsnP3kGzzzzV8LhLrq72zj22JV84AMraG1tYdWqCzjppFMA\nOOCAZVx22VVcccUXKCsr46677mX16ptZv/5lwOyi+eY3v0NHRzvnn/+fvPe9H8h913e+czennPIp\n3v3uo3jppRe47757uPnm1YP+HzU2zuLqq69n06a3WL36Jh544CHuuusbhMNhWltbCIUqWLnyI6xd\n+yhXX309LpeLvXvbeOCBh3C5XFxwwVlcd91NLFiwkN/85jEefvhHHHTQIezatYPvfvcB4vE4F154\nNkce+U5OOOEEWlq6eOWV9QQCgVwZfD4/PT3dBeXq7Oygq6uLO++8h9/97jfcc89d3HjjV0b+SyDE\nFDWcsHBgdledCFyklPIB+9QNpZQ6CThVa33GAOuuBM4AMsAarfWjVrfXT4FKIAH8p9Z6x76U4ZMr\nFg/aCqitLaelpWtfPn5ImzZt5OWXX+JPf3oSgK6uMD6fn0suuZLbb/8akUgPK1d+ZFiftXXrFt71\nrvcAMHfuPObOPYM77vg6RxxhhoPP56epaQE7dmzPvecznzmbH//4QS699GJqa+s44IDlVFUF+etf\nf8DTT/8Fn89PKpXKbb906X4AlJcHaGpaYC0HSSTMMYlDDjkMu91OVVU15eVBOjo6Cvb1Jz/5bx5+\n+EcAOBxD/8odfPBhACxcuIi2tjZsNhsrV36EP/7x9+zcuYPjjz+x33saG2fhcrkA2LLlbe644zYA\n0ukUc+bMY9OmjWj9L1atugAwu512797JwoWzAPD7/UQivV1akUhPQXgABIMhjjrq/QAcddT7c/sj\nxEwxnLD4MbALeFZr/Xel1BvAfaP9QqXU3cCHgPUDrKsALgUWYwbSeuBR4HzgH1rrryilzgKusbab\nUmw2O4aRYf78JlauPICVKz9Me/teHn/8MVpbW9H6DW699b+Ix+N84hPH8aEPfRSbzYZhZAb9zPnz\nF/DGG6/zvvcdzY4d27n//u9y4IEH8eqr6/jABz5IJNLDW2+9xaxZs3LvefLJ3/LRjx7PqlWX8ZOf\n/Ddr1z5CJpNg+fKDOOmkU3j55Zd47rln8sptG3K/tP4XYI7J9PT0UFlZmVs3b14Tp59+JgceeDBb\ntmxm3bp/DPlZb7zxGitXfpi33tqY66467riP8ZWv3EgsFuWii1YBYLfbyWQyuf/X3u+bzw03fIWG\nhgZefXU9bW2tuFwuDj30CK699ktkMhl++MMfMHv2nNx7mpoWsH37NsLhTrxeH+vXr+P00z9TUK6D\nDjqE559/lv32259XXnmZpqaFQ+6HENNN0bDQWt+plLpba522Xnqv1rptyDcN7W/AY8CFA6zrAbZg\nBoUfs3WB1voupZTD2mYe0DHAeye9yspKkskUkUiEv/zlD6xd+4jVTXQB1dXV7N3bxkUXnYPdbue0\n087E6XRywAHLue++e2hsnJ07qs934oknc+utX2HVqgtIp9NceumVLFq0hK9/fTUXX3wu8Xicc845\nn8rKqtx79t9/Obfdthqv14vNZuOaa75ENNrBl798C3/605MEAgEcDgeJRGJY+7V3bxuXXnox3d3d\nXHnltTgcjty6z3/+Uu644zYSiQTxeIxLL71qyM/atWsHl1xyEclkgquvvh6A2to6fD4fy5YdiNNp\n/souX34Qq1ffzDXXfKng/VdeeR2rV99EOp3GZrPxxS/eyNy581i37h987nPnEY1GeP/7P4jP58+N\nWZx44smsWnU5V1zxBTKZDMcd9zFqa+sIhzu57bbVrFnzDT772bO57bbVXHjh2TidTm644ZZh/d8I\nMV3YDGPogVyl1PHA+4CvAi8CtcDNWuvvFHnfucDlfV4+W2v9olLqaOAirfVpfd7jAn4EfBCz++tW\nrfU389b/GTgQ+A+tdb+WSb5UKm04nY6hNhEl8Mgjj7Bp0yauumroENhXF154Iddffz3z588f0+8R\nQjBgV8JwuqFuBj4DnAa8AHweeAoYMiy01g8AD4yoiPARoBHIHkL/Xin1rNb6BeszVyil9gOeABYN\n9UHt7ZERfnWvsR6zGI3/+q/b2Lx5U7/X77jjW3g8Zfv8+aPd566uGJFIYkTvvf76qwmHOwteCwQC\n3Hbbnf22jcdjXHzxeRx++JH4fFUl/blMxp/zWJtp+zzT9hf2fZ9ra8sHfH04YYHW+l9KqVuBh7TW\n3Uop96hLMrR2IArEtdaGUqoDqFBKXQds11r/BOgG0kN9yHR01VVfnOgiDOijHz1hxO9Zs+Ybw97W\n4ynjwQcfGvF3CCFKazhh0ayU+jZwJHCmUuoOYGspC6GUugLYqLVeq5Q6FnheKZUBngH+ALwC/Mjq\n2nIAZ5fy+4UQQgxtOGFxOnAScJfWukcptQn48r58qdb6KcyurOzzO/OWb8bs+srXDHx4X75TCCHE\n6A0nLLqBAPB1pZQT+AvmWUtCCCFmiOGExe3AEuBBzFHyszEHoC8bw3IJIYSYRIYzkeBK4GSt9Vqt\n9a+BUzAvqhMjJLPODm0yzzp79913sGrVBaxadQFnnPEJLrjgrJJ9txBTwXDCwklhC8TJDDwbqRRk\n1tnSGe9ZZy+99Eruuef73HXXvfj9Aa699oaSfLcQU8VwuqEeBp5SSv3Men468LMhtp8SHtn4G9bt\n+eeA6xx2G+nMyGedPbTuQE5efPyg62XW2ak762zWr371c97xjnexaNHgswsLMZGKXWg9WsOZ7mON\nUmod5rTkduBrWusnxqQ005zMOjt1Z50FSCaT/PrXj3D//T8e2Q9eiGEyMhky8TiZaIRMJEI6EiET\njZrL1muZaOHrmWjUfG6t2xwKMu+Wr2Ev85a0bMO9KO93wO+yz5VS92qtP1fSkoyzkxcfP2grQGad\nlVln+846C/DSS3/nkEMOG3CdEABGOt2v8k5bj5lItH+Fn63srfWZaARG2DKwOZ3YfT7sfh+umhrK\nm+Zhc3tKvm/DCosBnAlM6bCYCDLr7NSddRbgpZdeyAWzmH4Mw8BIJHor+2j/o/j+y4XbGCM8qQPA\nXlaG3evDWVmJY/Zs7F4vdq8Pu8+Hw+s1g8Dnw2G9Zvf6cPiy23ixuwon1Birg93RhsXQtYcYkMw6\nO3VnnQWzFffhDx83rP8TMf4Mw8CIx8yKu8fstmnbZBDe3dZ7RB+JkI5GBz3SJz3Cc3fsduxeLw6f\nD2cw1KdSN18vWPbmVfxeL3avF5tjakx4WnTW2YEopcJa6+AYlKdkWlq6Rj3KI5OPDd9vf/s4W7Zs\n5uKLvzAGpep1zTWXccklVzJnztySfab8nCeX7JF9rnsmEiEd6Snsr887ok/39PTr0hlxF47L1b8y\n9/r6V/LZZa+/4HWbx1O05T3eSjCR4MhmnVVK/QUY6H/eBpR25EQUNdazzo6n0c46W8qgEGMjk0gU\ndtFklyM9/fvxI5HCYIiO/Mje5nZj9/lwhkLYGxqtit2s0B0+H+U1FcQMp9ldkw2BvCN/uzXWJYob\ntGWhlPrAgCssWuunx6REJSIti5GRfZ4Ziu2zkUr1r9zzjuyz6wqO8PO2M/JOjBiO7JF9ruvG11uh\nO/z+3m4bX35/vTmY6/D6sDmH7kmXn/Go3j+ylsVkDwMhRHFGJkOypYXErp0k9zQTySToauso7NLJ\nq/iNYY5T5TgcuaN5V3VNYYWeX8nnh0Dea30HZ8XkNdoBbiHEJJJJJkjubiaxayfxXTtJ7NplBkTz\n7qGP9u323JG9s6ICh89vdtkMWLn7C0PA68Pmdk+6PnsxNiQshJhC0pEIibwwyC4nW1v6De7aPB7c\nc+bibmzE0zgLV30D1XPr6UraciEwGQdoxeRUNCyUUu/v85KBeTe7jVrrjgHeIoTYB4ZhkO7szIVB\nPBcMu0h39v+TcwTK8S5egrtxFu5Zs8zHxkaclVX9giBUW05ihvXhi9IYTsviJuAI4E+YZ0IdDWwG\ngkqpG7XWU36eqPESj8d58snfccIJHx/2e9avf5lAoJzFi5fs8/evWnUBV199PfPnN+3zZxVzyikn\n8PDDv8LjGf6VpA888D2qq6v5+MdPKbrt00//hWXLllNTU7svxcx55pm/8sMf/gCHw8Fxx32Mj33s\npIL1u3fvZvXqmzAMg2AwyM03f42ysn07C83IZEi2tha0ELLLmWi03/bOqmp8y5ZbYTAr12JwlA98\nz2QhSmk4YWEDDtJabwVQSs0C/hszNJ5iGkwqOF6ys86OJCyeeGItxxyzsiRhMZ388pc/o6np+pKE\nRXbW2fvv/zFer5eLLz6X9773/VRVVee2+cUvHmbFiv/g5JNP5Xvf+w6/+c1jnHLKacP6/EwySbJ5\nd0EYxHftMscTksnCjR0O3LV1uPc7AHdjY28wNDRg38dwEmJfDCcsZmWDAkBrvVMp1ai1Diulpmxn\nZ8svf07XSy8OuG6Lw046PfgUG4MpP+JIak8dvAKRWWen7qyzS5Yo9uxpBsy5o+rr6/uVPR2NDtBK\n2EWyZU//8QS3u6CFkFuuqyt6OqgQE2E4v5XPKqV+ijlVuR04DXhOKXUc5i1XxTDJrLNTd9bZ2to6\n7rvv2/zhD/+PRDzO6e87mo6//CkXCPFdO0l39B9PsAcC1nhCI+6GWbhnmcHgrKzCZh/O7WSEmByG\nExYXWf8uAFLAH4H7Me+g13+2tSmi9tTTBm0FyKyzMuus3+8n0bInFwZ333s35y1ayv6pNOv2NPOV\nL13DZXObcu9xVlVZ4wmNBS0GZ/mknhVHiGEbzv0sUkqppzDHLhzAc1rrFPDbMS7btCOzzk6+WWcz\nySSNLhdbN77J5l/8HHtbKy/87gne9eqrbObh3OeWRaM49+zB0bSAxtpa4q+up+Hc863xhEYZTxDT\n3nBOnf0M8GXgMcxuqEeUUqu11g+OcdmmHZl1dmJmnZ0zew4v/e1ZLjrrDCI93bxjXhPtD/6AX7+6\njq72do4OVXKKx8sN3/8OhmHwvupa6uc1Ea+q5Psvv8TNn7+Uq9IpvvOTH/JEx14Mw+Dar32doNXi\nEmImKDrrrFJqPXCM1rrNel4DPKW1Xj4O5Rs1mRtqZKb6rLNGJkOqo4PknmYSe5pJNvc+Jlv29D/r\nCHCWB3DWN+ZOQc11HVVVT9vxhJn2uz3T9hcmYG6oPI5sUABorVuVUiM/VUjsE5l19khmz55DqqOd\nRHOzGQr5jy17BpzXyF5WZoZAfT2u+npctfW55YYFs2htlXM0hBiO4bQsHgJagQesl84FqrXWk3pw\nW1oWIzNZ9tkwDNLhzlwQJPfsIdG822ox7BnwTmQ2jwd3nRkAvY8NuOrqcASDg465TJZ9Hk8zbZ9n\n2v7CxLYszsccs3gQc8ziT8DFoy6JmPEMwyDd1WV1Fe22HvdY4dBMJhbr9x6b242rzmoV5D/W1eMI\nhWR+IyHG2HDOhooC1+a/ppQ6HblyWwzBMAwy3d154weFoTDQdBY2lysXAAWthPp6HKEKCQQhJtBo\nLxX9HhIWAkgXBELe455m8zaXfdicTlx1dbjUflbroCHXSnBWVEzbgWUhSi1jZOiMh2mJttEabcs9\nBnxeTmn6OA57ae/tPdqwkEO8GSQd6SkMg+zA8p5mMj09/ba3OZ24autwLVma1zpoMAOhslICQYhh\nSmZS7I3utYJgb0EotMb2ksr0v1dJlbeCj89P42ByhMWoB49nssk862yqp4fY5s153UW9rYRM9wBn\nDDkcuGpr8S5a3G8swVlVzamfPJGHb5g+s87u3LmDr33tyxiGQUNDI9dc86V9nnVWCIBoKtobBpG2\ngpZCR7wTY4Dq1uv0MstfT423mhpvNbV5j4vnzKattf9B3L4aNCyUUjcNssoGyL0QR2EyzTqb7u6m\n5/UNRDZsIPLG66Ta9/bfyOHAVVODd8FCXH3GEVxV1dgcpT1yGYnxnnX23nvv5sQTP8HKlR/m8ccf\n4+c/f4izzjpvn79bTH+GYRBOdPW2CKwgyD7vSfbvrgWo8IRYVNHULwxqvNX4Xb5Bv89uG5uW+1At\ni6G6mm4tdUHG29/+/Bab/rVnwHV2h53MKGadXbhfHe9ZsWjQ9RM56+yKD6wgE43S/ocn0W+8xkOv\nvIzDZsNtt/OFpQcQOHAZ9778MhHDoD0a5ePHn8gpn/4sX7jscyyureXt55+Z0bPObt78NtdcY87B\ndeCBB/Otb/W/JkTMXOlMmrZYe0EYZLuNWqNtJDL9Lwp12BxUl1Uyv3wutb7CMKguq8LtcE3Angxu\n0LDQWt8yngWZCcZ71tnrLr6Elpdf5JJv3ErVr35BfOsWOhMpXgh38J5Fi/nECSfxaixGzVHvxe5I\ncdwhhxfMOnvqf54DyKyzAIsXL+XZZ//KRz5yPM888zSxWP+zucT0FkvFaYvtpSXSmtdKMMcT2uMd\nZAaYw63M4aHOV5sXBFW55cqyijFrBYyFGTtx/ntWLBq0FTBVZ52dXVvHR9UB3PXdb6EMg7evuxqA\nBqDV7sBkXhZEAAAZiUlEQVRZWUndGWey6vAjeegXP+Urv3+C2to6Dj56BVVVFTLrrCUS6SkID4BV\nqy7nm9/8Ok88sZZ3v/soQqGKIfdDTD2GYdCd7Ol3dlF2uSsx8NX+QXc5TcF5/cKgxltNwOWfNqd8\nz9iwmAilnnXWyGSYHQyx7vFf0/T882x5fQOPNu9isdfHG+kM7zl2JbbFS2i+/16OWH0rv77xi/gO\nWMYf//pnmXXWmnUWoKlpAdu3byMc7sTr9bF+/TpOP71wgoIXX3yeCy/8PPPmNfGznz3EkUe+c8j9\nEJNTOpOmPd45YBi0RtuIpweYMsZmp8pTwezKJdT4CscPqsuqKHMO/ySOqWxEYaGUOl5r/ZuxKsx0\nV4pZZ1OdnURef42e1/5J5LXXOLizgwd37+DmZBI8Hi44/uPs/4EP8u1HfsHq114l/vKLnHPeRTN6\n1tm5c+exbt0/+NznziMajfD+938Qn8+fG7M48cSTWbXqcq644gtkMhmOO+5j1NbWEQ53ctttq1mz\n5hvMm9fELbfciNvtoqlpEVdeeW2/souJYRgGyUySSCpKJBklkooStZbZm2Jzy65cKLTG9g7YXeS2\nuwpaBPnLVWUVJb9mYSoqOjdUPqXUy1rrw8awPCUzXeaGMlIpohvfpGfDP4m8toH4ttwdbnGEQvgP\nWI5v+XJ8ByzbpxvtTPVZZ0djMv2cx8tk3eeMkSGaiuUq+UjKqvTzl1MxIsmI+ZiKEklFiCbN5bSR\nLvodAZffCoIqar01BcEQdAemTXfRRM4NlW96/G9OYoZhkNzTTM9rG4hs+CcR/S+MeJyf7N7JzkQc\nu9eHw+/H4fdji0e548zPzJhZZ0sZFKL0EumkWdnnHdkPq9JPRoml+88HNhSHzYHP6cXn8lLtrcLn\n9OJ1luFz+fKWvTRWV+NKeKnxVuN1Tq2/k8lmpGGxdkxKMcOlIxEi/3qDyGsbiLy2gWRrS26dq6EB\n/7IDuXbZcnxqP+wjuMhtPHz0oyeM+D1r1nxj2Nt6PGU8+OBDI/4OMXIZI0MsFS+o7LOVf98K33we\n6902FR3wauKhlDk8eJ1eqr2VVgXvNR9dZWYQOH25Sj+7zucyH11217BaApO1JTUVjSgstNY3l+JL\nlVInAadqrc8YYN2VwBlABlijtX40b91+wN+Beq31yA5FJhEjkyG+ZbPZenhtA9G3NoI1WGv3egkc\ndji+ZQfiX74cV3XNBJdWTGWJdII9kVb2RFtpibRi7EjRFu4kYh3VR5OR3HIsFRvwauHB2G12q1L3\nUumpyFXk2SN8r7OsNwRy68xlr6NMxgGmmHE/G0opdTfwIWD9AOsqgEuBxYDf2uZRa10QuAPof0OD\nKSDV0Z4Lh57XX+udQsNmo2zBAjMcDlhO2cKFE3pltJh6MkaG9lgHeyKtNEdaaI60sMd6bI93DPle\nt8NtVfYhvP4GfK6y3iN6p3fISt/jcE+bfn5R3EScOvs3zPt5XzjAuh5gC2ZQ+DFbFyilbMD3geuB\nXw/nSyorfTido690a2vLR/1egEwiQfj1N2hft56OdeuJbOkdmHZXV1HxrhVUHnoIoYMPwlW+b99V\nKvu6z1PRVNrn7kQPu7r2sDPczM4u89+urj3s6t5DMt3/CuFKb4hldUuZVV5PY3k9jeV1hDzl+Nxe\nAi4fPpcXZ5HrXqaDqfQzLpWx2Ocx+01RSp0LXN7n5bO11v+jlDp6iLduA14HHPROK3Iz8ITW+hWl\n1LC+v7194PlWhmM0/ZyGYZDYtYvIa/+k57UNRP+tc7f5tLlc+JYtx79sOb5lB+KeNcu8fgLoiAGx\nie9TnYl9u5Nxn1OZFK3RvbnWQbaF0BxpoTvZf3I4j8NNg6+Oel8tdb5a6r011PlrqfPWDnj+f221\nuc/xGMSZ/lehT8af8VgrwdlQA74+ZmGhtX6A3luxDtdHgEZggfX890qpZ4Ezge1WADUATwLvL1VZ\nRyvd00Pkjdfo2bCByOsbSO3tnYzPPWu2GQ7LD8S7ZCl2t8y9KEzZieXyu4yyodAWa+93HYANG9Xe\nKuYH51qhUJMLh5B78NvGClFKk60N2g5EgbjW2lBKdQAVWuvF2Q2UUpuBlRNROCOdJrb57dw1D7G3\nN4F1nYrd5ydwxDvwL1+O74DluKqqinyamO7i6UQuBPYUBEMrsXT/obeAy09TcF4uDLL/qr3VuOyT\n7U9VzDST4jdQKXUFsFFrvVYpdSzwvFIqAzwD/GEiy5bc20Zkwwbziuk3Xu+9+5vdTtmixVbX0nLK\nmhbITX1moIyRYW+sneZIa0GX0Z5ICx3xzn7bO+1O6rw11PlqzG4j61+dr3bIaaeFmGgjuoJ7Khnt\nFdyZRAL37i3s/NsLRDZsILF7V26ds7oa/7ID8S1bjm///XH4/CUr70STvt2hdSd7cmcb5bcSWqJt\nA15fUOmpKOguyj5WTfBMozPt5zzT9hcmzxXc0962228lvvltAGxuN/4DD+q95qG+QfqHp7FkJkVr\ntM0Mgh4rEKLm40A3qClzlDHb31gQCnXWmILHIWNUYnqRsOgj9J6jcB12MLYFSylbvAS7a3LdgETs\nm1gqRkc8TEe8k3WdXby1Z3suHNpi7f0uSrPb7NR4q1gQnJ/XZVRDna9uWs0nJEQxEhZ9VKw4dkY2\nXae6jJEhnOii0wqCbCDkP++Mdw44sAxQ7gqwMNRkBoK/ljqv2Vqo8VbLlcZCIGEhpoBYKk5n3wBI\nFD4PJ7oGnHo6y+/yUe2tIuQJUuEOUeEJsrB+Dt50gDpvLT6Xdxz3SIipR8JCTJiMkaEr0V1w5D9Q\ni2CoGUmdNgchT5Cm4DwqPEEqPCEzEDwh61+QkDuIa4D7GUsLUojhk7AQYyKeTliVfm8A9A2Eoq0B\np4+qsoreSt96NAPBXJ5Ot60UYjKTsBAjYrYGeqxKv38AdCTM59HU4K0BR641MLcgACrc2UAwWwfu\nAVoDQoiJIWEhckrRGshOV90UDA7YIqjwhPC7fBN6rYEQYuQkLGagjJGhJdLKpvBWNnduYUt4G23x\ndvOexYOw2+yE3EHml88dYGygNxDccn2BENOShMUMEElG2RzeytvhrWzu3Mrm8FYiqd5gcNmdNARq\nmV8+tyAA8gMh4PJLa0CIGUzCYprJGBl29TTzdueWXDjsjuwp2KbWW82y6v1ZEJrHguA8Zgcaaaiv\nkDODhBCDkrCY4roS3WarodNsOWwJbyWeTuTWexxuVOViFgTn0RSaR1NwHuXuwASWWAgxFUlYTCHp\nTJrt3TtzLYa3w1tpjbYVbNPgr2dBcF4uHBr99dJ9JITYZxIWk1hHvNNqMWxhc+dWtnZtJ5k3w6nP\n6eWAamWFw3zmB+fKlchCiDEhYTFJJNNJtnXvyHUnvd25peB+CDZszA400hSax8LgfJpC86jz1sgF\naUKIcSFhMQEMw6At1s5maxD67fBWtnftJG2kc9uUuwIcVLMsNwg9t3zOgPdUFkKI8SBhMQ5iqThb\nu7azuXMrm6wupa5kd269w+ZgTvksszspNJ8FwXlUlVVKq0EIMWlIWJRY9oK3bFfS2+Gt7OzeXXCf\nhEpPBYfWHcTC4DyaQvOZG5g14ER3QggxWUhY7KNIMsqW8LZci6H/BW8uFoaact1JTaF5VHhCE1hi\nIYQYOQmLERjtBW9y8xwhxFQnYTEEueBNCCFMEhZ9bGh9g3++9Rr/2vOWXPAmhBAWCYs+/nfj4+yJ\ntMoFb0IIkUfCoo9LD70Qf8iFM+qVU1eFEMIifSh9VHhCzCqvl6AQQog8EhZCCCGKkrAQQghRlISF\nEEKIoiQshBBCFCVhIYQQoigJCyGEEEXJdRZCCDGFZAyD7miSrp4E4UiScE+CcCRBVyRBuCdBdaWf\n4945F3uJT/+XsBBCiAmWTKUJ9yQJWxV+bwD0hkF2fVckgWEM/ln+MifHHjoLr6e01buEhRBClJhh\nGPTEUrmj/VwLoMes7Dt7rCCw1scS6aKf6fU4KPe5qasMEfS5CfpcBP1uyn1ugv7e50sX1NAVjhb9\nvJGSsBBCiGFIpTNWZZ/XAsi1BJJ5wWBuk84McfgP2G02yn0uakJeQn4X5X63GQJ+N+U+V2456DOf\nu13Du9VBmcdJVyl2uA8JCyHEjGQYBtF4Ou9IP68VYC139STojJjjA5F4quhnelwOgn4XTQ3lvUf8\nfldeCPS2AvxeV8nHFcaShIUQYspKpTMkkhniyTSJZNp6NJ/Hk2lcWzrY2RwesN8/3JMklc4M+fk2\nIOBzURn0MN9Xbh7x57UAgj435X4XIZ8ZBB739L3RmYSFEGLMZDJGruI2K/NMn+fpfpV9PJEhkcou\nW6+nMiQS1vapDHFruVhXz0BcTjtBn5u5dQGCPrP7J5Q76rdaAVYYBLwu7Papc/Q/liQshJjBMhnD\nqpitCjuRJp6yHvtW4gNU7PlH8f3XZ4oeuQ+XzWZ28WT/+YNOPC4Hbuu522XPrTNfM5/XVPshnSkY\nAPa4HDKr9ChMSFgopU4CTtVanzHAuiuBM4AMsEZr/ahSygZsB960NntOa33duBVYiEkumcrQFSkc\nfM0uZ8/H74okSGUMIrFkrlJPpkpUmQNutwOP047b5aCi3GNW3E6rEnc7cDutytxtt9aZr2crdnde\nGLj7vOZ02EZVwdfWltPSMhbDvTPPuIeFUupu4EPA+gHWVQCXAosBv7XNo8Ai4GWt9QnjWFQhJkwm\nY9AdSxZU9OE+y7kwiCSIxoufeuly2vGVOXE57NYRdm+FnH80nl9pe1z2vKP3wuf5lbrLaZej9Wlu\nIloWfwMeAy4cYF0PsAUzKPyYrQuAw4HZSqm/AFHgcq21HoeyClEShmEQS6StI33r6L/PUX9+S6Ar\nmhzywivoPfWyOujN9bVn+93Lfb2Dr9lTLz0uB3V1QTnSFqMyZmGhlDoXuLzPy2drrf9HKXX0EG/d\nBrwOOIBbrdd2AbdqrX+plHov8BBw5FDfX1npw+kc/ZkJtbXlo37vVCX7PDLJVJrO7gQd3XE6rX8d\nXQnzMf+1bvO14XT5BLwuQgEPcxuCBP1uKgIeQgEPFQE3ofLssvk42sHXmfZznmn7C2Ozz2MWFlrr\nB4AHRvi2jwCNwALr+e+VUs8CLwEp63OfUUrNUkrZtNaDHnu1t0dGUWrTTOznlH0eu66foM/NnFp/\nv6P9wiN/8+jf6Rje3J7xSJx4JL7P+zzdzbT9hX3f58GCZrKdDdWO2c0U11obSqkOoAK4GWgDbldK\nHQxsGyoohMhkDCLxFD3RJN2xJD3R/OUkPbEUybRBS3ukJF0/5bmpF6zXrLNv5MwbMV1MirBQSl0B\nbNRar1VKHQs8r5TKAM8AfwBeBB5SSh2H2cI4a8IKK8ZVOpOhJ2ZW9D3RVEFlbz72Lndnn0dTw7ra\nNsvncVLud1Nf5Suo6POvts0u+8qcU+qqWyFKxWYUO5Saolpauka9Y9J0Lb1UOmMd2edV8tGUVdlb\nQTDA68Pp6slyOuwEvE78Xhf+Mhf+MicBr8t6br4eKOtdbppbSSKaGHbXz3Qw0363Z9r+Qkm6oQY8\nGpoULQsxdSRTabqzlXneEX53fiXf58i/O5YiPoxZNbPcLjv+MnOCtWzF7i9z4fc6zco+W/mXuQrC\nYLgTrWVVh7y0JIbfAhFiJpOwmOFS6Qxbmrv4964udjWHC7t3rMo/PxwSI7iIq8ztwF/mor7Sa1X2\nLgJDVf7Wo2sfzmITQowNCYsZJhJLsXFHJ29u7+DN7Z28vStc9JROr8eJv8xJY42/T2Xfv/L351X+\nM6l7R4jpTsJimtsbjvHm9t5w2L6nm+xgjg2YWxdg8ZwQi+ZWYqTTBZV9wOvCV+bEYZdKX4iZTsJi\nGskYBrtaewrCobUzllvvctpZOreCJXMrWDonxMJZIXxl5q/ATBwIFEIMn4TFFJZMZdiyu4s3t3fw\n720dbNzRSU+sd8A24HVx6JIalsypYMmcEPMbyqVrSAgxKhIWU0gklrTGGzp5c1sHm3Z1FUwBXVtR\nxsGLa1gyJ8SSORU0VPvkmgAhRElIWExie8Mx/m11J725rZMdLXnjDTZzvCHbalgyp4LKcs+EllcI\nMX1JWEwSGcNgZ/54w7YO2sK9c/+4nXbUvAoWz6lg6dwQi2aF8HrkxyeEGB9S20yQZCrD5t1h/r3N\nbDls3N5ZMEVFwXjD3BDz62W8QQgxcSQsxklPLMnG7Z25lsPbfcYb6iq8ZjjMNbuVGqp8MgGdEGLS\nkLAYI22dsdzpq29u72BHS0/BeMO8unJzrGFuBYtnh2S8QQgxqUlYlEDGMNjZ0pMLh39v72DvAOMN\nS+ZUsHRuBQtnBWW8QQgxpUiNNQrJVJq3d3XlwqHveEO5z8VhS2tzZynNqw/IeIMQYkqTsBiGnliy\n4KrozbvCpNK9M6DXVXo5dGnvxW8y3iCEmG4kLAawZ2+E51/bXTDekGWzwbx6c7xh6ZwKFs8JURGQ\n8QYhxPQmYdHH3b98hVfeass9d7vs7D+/MjcYvbBRxhuEEDOP1Hp9zKrx4/O5mV/rZ8ncCubWyXiD\nEEJIWPRx6gcXywysQgjRhxwyCyGEKErCQgghRFESFkIIIYqSsBBCCFGUhIUQQoiiJCyEEEIUJWEh\nhBCiKAkLIYQQRdkMwyi+lRBCiBlNWhZCCCGKkrAQQghRlISFEEKIoiQshBBCFCVhIYQQoigJCyGE\nEEVJWAghhChqxt78SCllB+4FDgbiwHla64156z8BfBEwgIe11ndPSEFLqNg+5233fWCv1vqL41zE\nkhvGz/ly4DygxXrpQq21HveCltAw9vlI4E7ABuwGztRaxyairKUy1D4rpRqAn+dtfgjwRa31feNe\n0BIaxs/508CVQBp4UGv93X35vpncsvg4UKa1fjdmKNyRXaGUcgC3AccC7wY+p5SqmZBSltag+5yl\nlLoQOHC8CzaGiu3z4cBntdZHW/+mdFBYhvrdtgH3A2drrd8L/D9g/oSUsrQG3Wet9e7szxe4DngZ\n8/9gqiv2u/1fmHXYUcCVSqnKffmymRwW2T8UtNbPA0dkV2it08D+WutOoBpwAImJKGSJDbrPAEqp\n9wDvBL43/kUbM0PuM2ZYXKeUekYpdd14F26MDLXPS4E24HKl1NNA1TQJyGI/52xQfhu42Pobn+qK\n7fOrQAgow2xF7tN0HTM5LIJAZ97ztFIq1y2ntU4ppU4GXgGeAnrGt3hjYtB9Vko1AjcDqyaiYGNo\nyJ8zZvfERcAK4L1KqePHs3BjZKh9rgHeA9yDedR5jFJqxTiXbywU+zkDnAC8Nk3CEYrv8wbgH8Br\nwG+01h378mUzOSzCQHnec7vWOpW/gdb6EWA24AY+O45lGytD7fOpmBXJbzGbtGcopc4a3+KNiUH3\n2TrSvEtr3aq1TgBPAIdOQBlLbaifcxuwUWv9htY6iXlk2u8ofAoq+vcMnAl8f/yKNOaG+t0+CDgO\nWAA0AXVKqVP35ctmclg8C3wUQCn1LuCf2RVKqaBS6mmllEdrncFsVWQmppglNeg+a62/pbU+3OrX\nvQ34qdb6hxNRyBIbdJ8xj8w2KKUCVnCswDwSm+qG2udNQEAptdh6/j7MI8+pbqh9zjoC+Nt4FmqM\nDbXPnUAUiFpdbnuAfRqzmLGzzuadSXAQZn/e2cBhQEBr/X2l1AXAuUASs+/vC1O9n7PYPudtdxaw\n3zQ7G2qwn/NngEswzyb5k9b65gkrbIkMY59XYB4Q2IC/aa0vnbDClsgw9rkW+IPW+pAJLGZJDWOf\nLwLOwRxvfQs432pBj8qMDQshhBDDN5O7oYQQQgyThIUQQoiiJCyEEEIUJWEhhBCiKAkLIYQQRUlY\nCDEKSqmnlFJHT3AZfjhNLpwUU4CEhRBCiKJm7BTlQvSllJoDPAz4Ma/YvwRz7qijtdabrZbEl62r\n3AEuUEplp/q+XGv9lFLqGOB2zEnb2oHTtdatSqmvAccAVUArcLLWerdSajfwOOaV1LswL7K6BJgD\nnKW1flop9RTwBuYkj2XAZVrrJ/uU/bPAZZgHgP8APj/Vpx0Xk4u0LITodS7mhGtHANdgzuo5lG6t\n9WHAfwI/UUp5gBuAi6zPeBw4zJpaYz/gPVrrpcBG4NPWZ9Rb37mf9fwkrfX7gC9jVv5ZHuu7zgB+\npJRyZ1copZYB51uffwjm1A5Xjep/QIhBSFgI0euPwFVKqZ9iTiB5T5HtHwDQWr+KefOk/YC1wKNK\nqXuAN7TWT1o3pLkSOE8pdQfmPVICeZ/zO+txC/DnvOX8uXzut75rPWYL5KC8dR8ElgDPK6XWAyda\nZRGiZCQshLBorZ8FDgB+D3wKs2VgYHYzAbj6vCV/VlMbkNRafxM4GrP1cLtS6ktKqcOBJzH/3n4F\nPJr3mfSZr6fvTKkDvW7v89wB/EJrfYjVsngH02+qeTHBJCyEsCilbgc+o7X+EWZlexjm+MIya5MT\n+7zl09b7jsCcwfZNpdTfgXKt9V3AN63P+ADwlHUbz9eBlZgV/EiclvddlRTOMPoUcJJSqs6aPfe7\nFHZhCbHPZIBbiF7fBn5qnY6aBi4GuoFvK6Vuxmxx5AsopdZZ256htU4qpa4HfqiUSmFOEX0R5kD3\nI0qpV+mdxXjBCMu2UCn1srX8Ka11WikFgNb6FaXULZhdWHZgHeasskKUjMw6K8QkZ50N9WWt9VMT\nXBQxg0k3lBBCiKKkZSGEEKIoaVkIIYQoSsJCCCFEURIWQgghipKwEEIIUZSEhRBCiKL+P7aCsz2o\nBK1kAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11fcbad90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch3_1.best_score_, gsearch3_1.best_params_))\n",
    "test_means = gsearch3_1.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch3_1.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch3_1.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch3_1.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch3_1.cv_results_).to_csv('my_preds_subsampleh_colsample_bytree_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(colsample_bytree), len(subsample))\n",
    "train_scores = np.array(train_means).reshape(len(colsample_bytree), len(subsample))\n",
    "\n",
    "for i, value in enumerate(colsample_bytree):\n",
    "    pyplot.plot(subsample, -test_scores[i], label= 'test_colsample_bytree:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'subsample' )                                                                                                      \n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'subsample_vs_colsample_bytree1.png' )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
